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ABSTRACT 

science, education, and mathematics faculty and how they convert ideas to 
action research proposals and plan to gather and analyze their research data. 
The study also described some of the research outcomes of successful action 
research projects. Rubrics were used on a diverse set of products and 
processes involved in the action research project by faculty teams at 53 
universities. These teams were participating in NASA Opportunities for 
Visionary Academics (NOVA) , a project aimed at improving science education at 
U.S. universities. The development of the rubric allowed quantification of 
approaches across research proposals. Further documentary analysis was 
conducted on course change proposals by developing a matrix that noted: (1) 

the conception of action research articulated; (2) research questions or 
research focus; (3) research design; (4) proposed data collection methods; 

(5) proposed data sources; (6) proposed instrumentation; and (7) proposed 
data analysis. Final outcome reports and conference presentations of 16 
action research projects were also analyzed. Data obtained from site visits 
to 20 of the institutions were used to triangulate the findings. It was* found 
that action research projects have produced a wealth of information about 
successful innovation in undergraduate courses, although many faculty are 
still novices in the implementation of action research projects. The majority 
used quantitative approaches over quality designs. The outcome of these 
action research efforts has shown significant gains in student learning, as 
well as in faculty professional development, although the latter was less of 
a focus in the cases analyzed. Observational data undertaken during site 
evaluation visits support the idea that faculty shifted their practices away 
from traditional didactic lessons, and that action research facilitated this 
process. An appendix contains a sample scoring rubric. (Contains 37 
references.) (SLD) 



TM 035 171 

Raudenheimer, Carol Dianne 

Faculty Conceptions and Practices of Action Research in the 
NOVA Program. 

National Aeronautics and Space Administration, Washington, 
DC. 

2003-04-00 

20p.; Paper presented at the Annual Meeting of the American 
Educational Research Association (Chicago, IL, April 21-25, 
2003) . 

Reports - Research (143) — Speeches/Meeting Papers (150) 
EDRS Price MFOl/PCOl Plus Postage. 

^Action Research; ^College Faculty; ^Educational Innovation; 
Higher Education; Mathematics Teachers; ^Science Teachers; 
^Scoring Rubrics; Teacher Educators 



This study examined approaches to action research held by 




Reproductions supplied by EDRS are the best that can be made 
from the original document. 



TM035171 ED419 3<^8 



Action research in the NOVA program 



1 



Running Head: ACTION RESEARCH IN THE NOVA PROGRAM 



Faculty conceptions and practices of action research in the NOVA program 



Carol Diarme Raubenheimer, Educational Studies 
Brescia University, Kentucky 



Paper presented at the aimual meeting of the 
American Educational Research Association, 
Chicago, IL, April 23, 2003. 



U.S. DEPARTMENT OF EDUCATION 
Officoof Educational Rosoarch and Improvement 
EDUCATIONAL RESOURCES INFORMATION 
/ CENTER (ERIC) 

Ipr This document has been reproduced as 
received from the person or organization 
originating it. 

□ Minor changes have been made to 
improve reproduction quality. 



• Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 



C._Raubenh.elimr 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 

1 



Acknowledgements: This work was in part supported by the NOVA (a higher education 
faculty development program) funded by the National Aeronautic and Space 
Administration - NASA. The views expressed are those of the authors. 










2 



Action research in the NOVA program 



2 



Abstract 

This paper examines (a) approaches to action research held by science, 
mathematics and education faculty, (b) how they convert ideas into action 
research proposals, (c) how they plan to gather and analyze their research data, 
and (d) describes some of the research outcomes from successful action research 
projects. 

Rubrics were used on a diversity of products and processes involved in the 
action research process by faculty teams at 53 colleges and universities. The 
development of the rubric allowed quantification of approaches across research 
proposals. Further documentary analysis was conducted on course change 
proposals by developing a matrix that noted (a) the conception of action research 
articulated, (b) research questions or research focus, (c) research design (d) 
proposed data collection methods, (e) proposed data sources, (f) proposed 
instrumentation, (g) proposed data analysis. Final outcome reports and conference 
presentations of 16 action research projects were also analyzed. Data obtained 
from site visits to 20 of the institutions data were used to triangulate the findings 
from the analysis of action research projects. 

It was found that action research projects have produced a wealth of information 
about successful innovation in undergraduate courses, although many faculty are still 
novices in the implementation of action research projects. The majority used quantitative 
approaches, over qualitative designs. The outcome of these action research efforts has 
shown significant gains in student learning, as well as in faculty professional 
development, although the latter was less of a focus in the cases analyzed. Observational 
data undertaken during site evaluation visits support the idea that faculty shifted their 
practices away from traditional didactic lessons and that action research facilitated this 
process. 
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Introduction 

NASA Opportunities for Visionary Academics (NOVA) is a NASA-fimded project 
aimed at improving science education at Universities across the USA by introducing 
innovative science and mathematics classes for pre-service teachers. New courses that are 
developed utilize the NASA enterprises or themes as a major focus and adopt an inquiry 
approach to teaching science and mathematics. The national network consists of more than 
76 Universities in 37 States. Courses are housed in a number of different academic units, 
61% in science departments, 9% in mathematics departments, 22% in science education or 
education departments, 1% in engineering units, 1 % in technology units, 1% cross listed 
(interdisciplinary) and 5% unspecified. 

To become involved in NOVA activities, a team of three faculty members (one 
from education, one from science or mathematics, and an administrator) attends an initial 
training workshop. Thereafter they develop a grant proposal, which is submitted to NOVA 
for funding consideration. Beginning in 1998, as a regular part of the NOVA Phase I 
workshops, action research formed one component of the professional development 
process. All proposals to NOVA included an action research component. The education 
faculty member on the NOVA team usually took on the role of assisting science and 
mathematics faculty members in the development and use of action research plans in 
improving teaching, student learning, and course design. 

Once funded, the team implemented the new science or math course, gathered 
action research data, and submitted a final report that included the results of the research. 
Each year, follow-up leadership development conferences are held for ongoing faculty 
development, additional project opportunities and the presentation of action research 
projects to the NOVA network. This context provides an ideal situation for research into 
issues relating to change and innovation in higher education because of the database of 
materials that has been accumulated and housed at the Science Teaching and Learning 
Center at the University of Alabama and the range of institutions and faculty members 
involved in the initiative. 
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Background and signiflcance 

Action research is an iterative process involving successive cycles of question 
generation, planning, action, observation, and reflection, with the latter being termed the 
four moments of action research (Zuber-Skerritt, 1992; Hopkins, 1993). The difference 
between this approach and traditional research is that it does not end with data analysis, 
but leads to the generation of new questions and new action (Cross, 1998). Action research 
is concerned with results that are relevant to immediate practice. 

Previous published research with university faculty (Kember, 1998, 2000) as with 
K-12 teachers (Cochran-Smith & Lytle, 1992, 1999; Arhar, Holly & Kasten, 2001), has 
validated the effectiveness of action research on the development of expertise in teaching 
and for enhancing student learning and achievement. Action research is a powerful form 
of professional development. 

In the US, classroom research has been advocated for many years as a mechanism 
for contributing to the scholarship of teaching in higher education (Cross & Angelo, 1989; 
Cross, 1998; Paulsen, 1999, 2001). Classroom research is essentially equivalent to action 
research (Bondy & Ross, 1998), with the purpose being to improve teaching and learning, 
which means that the cycle from question formulation to changing practice must be 
completed (Cross, 1998). Neither action research nor classroom research have been 
widely applied in higher education in the US, except for in some teacher education 
programs (Ross and Bondy, 1996; Bondy and Ross, 1998). In part this is because, 
historically, teaching scholarship has been synonymous with research publications and 
quantitative research (Kreber, 2000), and there has not been a focus on improved teaching 
practice for tenure and promotion review (Boyer, 1990). 

One of the main purposes of the NOVA project is to encourage faculty to adopt 
new approaches to teaching and assessment, and to the evaluation of their work. NOVA 
promotes a cognitive apprenticeship approach, which involves people changing roles from 
teacher to learner within an action research model, involving cycles of planning, action, 
observation and shared reflection (Sunal, Hodges, Sunal, Whitaker, Freeman, & Edwards, 
2001). This allows the teacher-learner to review the adequacy of traditional approaches 
and to make changes. Ownership of change is a central concept. Thus, the NOVA project 
has attempted to provide an alternative research framework to the dominant paradigm, and 
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to bridge the gap between research and practice by promoting research on classroom 
action. 

After Habermas, Carr and Kemmis (1986) categorized action research projects as 
serving one of three knowledge-constitutive interests, (a) technical, (b) practical, and (c) 
emancipatory, with each interest serving a particular purpose. These interests can be 
related to differing approaches to research, where different researchers emphasize different 
aspects of practice in their investigations and rely on different research methods and 
techniques that seem appropriate in the study of practice viewed from the particular 
perspective they adopt. Most faculty approach their research from their particular 
discipline and the accompanying paradigm (Huber, 1999). That is, different 
methodological perspectives in research on practice are related to different epistemological 
perspectives on the nature of practice (Kemmis and McTaggart, 2000). 

Keller (1998) noted that the research on and in American institutions of higher 
education has been dominated by “methodological monism” (p. 267) where the scientific 
method has predominated over other approaches. He argued for more research in higher 
education using other methods of enquiry, particularly pluralistic approaches. Kezar 
(2000) added that much of the critique of research in higher education points at the need to 
bridge the gap between research and practice, and suggests that action research provides 
one way to achieve this. 

Kember and McKay (1996) added that action research is “the mode of research 
associated with critical theory” (p. 231) and that it necessarily is concerned with social 
practice, is participatory, allows participants to decide topics, aims toward improvement, is 
cyclical, reflective and involves systematic inquiry. These reflect the intentions of the 
NOVA program in promoting action research amongst teams of interdisciplinary faculty 
members. The analysis of approaches to action research adopted by faculty in education, 
mathematics and science education at a variety on university campuses across the US 
provides an opportunity to examine the extent to which action research has contributed to 
shifts in teaching and research practices. 
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Research Purpose 

The purpose of this paper is (a) to examine the approaches to action research used 
by education, science and mathematics faculty teaching content courses to education 
majors, (b) to consider how faculty converted ideas into action research proposals, (c) to 
analyze how they plan, gather and analyze their research data, and (d) to describe some of 
the research outcomes from successful action research projects. 

Method 

Rubrics as a form of alternative assessment are common practice in educational 
settings and have been used to assess a diversity of products and processes including 
reading, writing, portfolios, research projects, as well as knowledge and skills in content 
disciplines (Taggart, Phifer, Nixon and Wood, 2001). They are useful tools for assessing 
restricted as well as extended performance tasks (Burke, 1999). Rubrics can be classified 
as holistic or as analytic, with holistic rubrics being used to rate a few skills without in- 
depth analysis (Wilson and Onwuegbuzie, 1999). In contrast, analytic rubrics are typically 
used with extended performance tasks because there are several criteria for assessing 
performance. Thus, when constructing an analytical rubric it is necessary to break the task 
down into smaller components and to rate item each on a scale. 

Wilson and Onwuegbuzie (1999) created analytical rubrics for assessing graduate 
student performance in developing and presenting research proposals. Their rubrics were 
used as the basis for developing a four-point rubric for scoring action research proposals, 
where 1 = novice; 2 = apprentice; 3 = proficient, and 4 = distinguished. This rubric was 
used to rate the action research proposals of 53 institutions in the NOVA program. The 
rubric is an analytical rubric containing 14 items, with a total possible score of 56, which 
would assume a score of 4 on all 14 items. This allowed quantification of approaches 
across proposals. Each institution received a total score and was categorized as (a) novice- 
apprentice (score of 14 - 27), (b) apprentice-proficient (score of 28 - 41), or proficient- 
distinguished (score of 42 - 56) (see appendix 1). 

Documentary analysis (Merriam, 1998) was conducted on all 53 proposals by 
developing a matrix that noted (a) the conception of action research articulated, (b) 
research questions or research focus, (c) research design (d) proposed data collection 
methods, (e) proposed data sources, (f) proposed instrumentation, (g) proposed data 
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analysis. Documentary analysis was also undertaken on the abstracts of papers presented at 
the 2001 and 2002 NOVA Leadership Development Conferences. 

Site visits were made to 20 of the participating institutions and included 
interviews with all NOVA team members, interviews with students, as well as an 
observation of a lesson. Classroom notes field notes were written and sessions rated using 
the ESTEEM instrument (Burry-Stock & Oxford, 1994). Data obtained from site visits 
was used to triangulate the findings from the analysis of action research projects. 

Findings 

A total of 53 action research proposals were analyzed using a four point scoring 
rubric. The mean score was 36.07, with a S.D. of 8.21. Eight institutions (15%) were 
categorized in the novice - apprenticeship range; 32 (60%) in the apprentice to proficient 
range; and 13 (25%) in the proficient to distinguished range. So, in the first stage of 
proposal development, most institutions were still in the initial stages of developing 
action research plans because the majority of proposals were categorized within the 
apprentice-proficient range. A quarter of the institutions had clear and well-developed 
action research proposals that provided a comprehensive framework for meaningful data 
collection and analysis. 

All 53 action research proposals reviewed reflect the desire to move away from 
traditional lecture approaches to more interactive methods to enhance student learning. 
Action research conceptions were well articulated in 3 1 of these proposals and the cyclical 
nature of change is clearly reflected. Some explanations given include 'developing a best 
practice model', 'analysis and improvement on an ongoing basis', 'developing a plan for 
severe problems' and 'authentic feedback to determine student strengths and weaknesses, 
for future improvements for improved student learning'. This is consistent with an inquiry 
approach to teaching where faculty are concerned to find better ways to enhance student 
learning. The purposes elaborated for research were to enhance (a) student content 
knowledge, (b) student process skills, (c) student attitudes and/or (d) student teaching 
outcomes. These were specifically stated in 50 proposals, showing explicit concern for 
enhancing student learning. 

The documentary analysis of the 53 proposals reveals that most were weak on the 
data analysis component, with 39 proposals (74%) making no mention about data analysis 
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or simply stating that data would ‘be analyzed’ or ‘compared’ to other groups. Forty-one 
percent of institutions did not devise actual research questions or hypotheses, although in 
almost all cases these could be inferred. Many institutions (57%) did not specify the type 
of instruments that would be used to collect data, making broad statements about methods 
of data collection, such as by survey or questionnaire. 

Quantitative approaches were represented in most of the proposals, with the pre- 
test, post-test design being the most common. Twenty-eight institutions (53%) suggested 
this approach. Another quantitative approach commonly advocated was that of 
experimental and control groups (30% of institutions), in some cases in a true quasi- 
experimental design or combined with a pretest post-test design. In these instances the 
NOVA course was to be compared to a comparable class that was not being taught in the 
new way. Fourteen institutions (26%) made no mention of their design, while a few 
planned qualitative designs, such as case study, ethnography, or ongoing processes of data 
collection. Strong proposals, rated proficient-distinguished, included both quantitative and 
qualitative approaches, indicating that sound proposals could be either qualitative or 
quantitative. 

The quantitative approaches were usually to be combined with additional 
qualitative techniques and were designed to elicit more in-depth data, to provide a deeper 
understanding about teaching and learning. Sixty-two percent of institutions mentioned 
methods other than surveys to assess student outcomes. The range of techniques suggested 
for student assessment was wide and includes journals, on-line tasks, performance tasks, 
rubrics, observation, interview, concept maps, quick writes, journals, portfolios, discussion 
groups, self evaluation, peer review, and logs of class activities. This is an impressive 
array of qualitative techniques, which professors in science and mathematics disciplines 
have embraced. These alternative approaches provide valuable ways to assess student’s 
content and conceptual knowledge, skills and attitudes, but as already noted, there was 
insufficient discussion on how these would be used to gather data, and how that data 
would be analyzed. 

An interesting, simple and yet effective approach was suggested by SUNY at 
Oswego, where they planned to have student write anonymous ‘quick writes’ (Cross & 
Steadman, 1996) where students would address key questions like “What have you learned 
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this week?" “Are you having difficulty with a particular concept?” and “How can your 
instructor enhance your learning?” These were then collated each time by a different group 
of students who would collect, review and summarize the answers. These results formed 
the basis of a discussion on the progress of the course. Many other innovative examples 
could be cited, including the use of concept maps by SFU and multiple assessment 
strategies used by Morehead SU for each unit, where each unit was assessed in a different 
way, giving students exposure to different methods and instructors multiple ways of 
assessing students. 

It is important to note that most institutions saw the students as the primary sources 
of data. There were only eight institutions that mentioned faculty as a source of data, 
indicating that there is a strong focus on the students as the unit of analysis, rather than on 
the faculty member, teaching strategies, student-teacher interactions, or assessment 
methods. To this end, the bulk of the research questions focused on (a) student content 
knowledge, (b) student science/mathematic skills, (c) student attitudes to science/ 
mathematics, (d) student ability to integrate knowledge, (e) problem solving, and/or (f) 
teaching efficacy. Only 10 institutions had a research question or a suggested focus that 
pertained to faculty and their teaching, and mostly these were not actualized in the 
methodology for data collection and analysis. In three cases action research was 
conceptualized as something for ‘future teachers’ (not college professors) and was 
suggested as a component for students during their practicum or student teaching 
experiences. 

The action research project being conducted at Cleveland State University is one 
example of an action research approach in which the instructors are focusing on their own 
teaching. The two instructors (one from education and one from science) have co- 
constructed a 'teaching portfolio' that includes personal reflections, teaching philosophies, 
methods and activities, student surveys and suggested improvements for teaching. This is a 
potentially powerful tool that can lead to constructive reflection, course redesign and 
changes in teaching. California State University at Northridge focused on the role of 
scaffolding, the process whereby the instructor provides structured support to achieve 
student learning outcomes, as a vehicle for dealing with student misconceptions. The 
research was to include video and audiotape of classrooms, transcriptions of the lessons 
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and discourse analysis. Here the research focus was clearly on faculty, their classroom 
dynamics and selected teaching strategies. 

Outcome of action research projects 

Faculty who have engaged in quality action research projects have made 
presentations at recent NOVA Leadership Development Conferences (LDCs). At the 2002 
LDC 10 action research papers were presented by different Universities. One presentation 
by Cleveland State University (described above) focused on investigating effective ways 
to teach science for improved student learning, highlighting the way in which faculty from 
different disciplines can work together productively. Another paper by Stephen F Austin 
University highlighted the need for instructors to create a ‘friendly learning environment’ 
for students with poor attitudes because attitudes to science are related to the desire to 
learn conceptually. All of the other eight papers reported positive quantitative and 
qualitative gains in student conceptual knowledge, attitudes to science and/or science 
teaching efficacy as a result of the new teaching methodologies. 

At the 2001 LDC six action research papers were presented, including papers 
describing student outcomes as well as those discussing new approaches to teaching and 
processes for faculty change. Four of these highlighted improvements in student attitudes, 
content knowledge, process skills and teaching efficacy, and one institution discussed 
approaches to problem solving. Fox University presented a unique project called Science 
Outreach, where education and science majors work together for eight weeks during the 
summer with students who have been home schooled. Students are responsible for 
plaiming and implementing instruction. They report that many of the science majors 
subsequently shifted their career goals to education. Kansas State University described a 
valuable action research model for education and science faculty to work together. This 
model called ‘peer consultation’ involves cycles of pre-observation discussions, written 
critique of observations (by the education faculty member), and written interactive 
reflections and follow-up between the two professors. This provides valuable data for later 
analysis of action. 

Of the 16 action research papers presented at the LDCs, 13 describe improvement 
in science or math content knowledge, attitudes to the subject, process skills and teaching 
efficacy. The other papers dealt specifically with instructor issues such as the classroom 
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environment and teaching strategies. The studies of student outcomes mostly used a pretest 
- posttest design, and some made comparisons to traditional classes. These quantitative 
and qualitative projects clearly show improved student outcomes as a result of the new 
courses. Clearly the action research projects have produced a valuable set of information 
about successful courses and improved student learning. 

The outcome of these research efforts show significant gains in student learning, as 
well as in faculty professional development. Observation data undertaken during site visits 
indicate that faculty have shifted their teaching practices because only one lesson observed 
(5%) was a completely traditional didactic lesson. During faculty interviews many 
claimed that the new approaches are also being integrated into the other classes they teach, 
and into the classes of other faculty members who have observed their new strategies. 
Faculty also repeatedly stated that interaction between departments or disciplines was a 
valuable form of professional development. Faculty from education tended to be used as 
‘consultants’ providing ‘new’ ideas on how to teach and assess student learning. For 
people who had only previously taught in a traditional format these were exciting 
opportunities for them to review their own teaching processes. 

Classroom observations 

Site visit data of classroom observations of NOVA courses being taught shows that 
69% were using inquiry based lessons in which students were actively engaged in learning. 
These lessons were described by the observers as “an excellent learning cycle class 
involving an engagement activity, student discussion, mini-lecture, and a follow-up 
activity for application”, or “the NOVA model was fully implemented”, or “a fully 
developed lesson using innovative strategies and well integrated with the NASA 
enterprises”. In all of these cases the interview data shows active teams involved in design, 
planning and implementation, with faculty noting that the initial NOVA workshops had 
consolidated the teams and that NOVA had brought together different disciplines. Good 
collaboration has helped shift faculty from teacher-centered teaching to student-centered 
approaches. 

In the remaining 32% of classes, one course was an on-line course and so the 
lesson was Internet based, with little opportunity for student interaction, although the 
instructor did answer questions from students. In four of the classes (19%) there were 
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elements of student participation through the presentation of class projects, the 
construction of models, or the use of graphing calculators. A large portion of these 
sessions was dedicated to direct instruction without any student participation or 
interaction, indicating that some elements of student-centered approaches have been 
adopted, but that faculty were still experimenting with new approaches and shifting their 
views. At one of these institutions two of the team members were new and had never 
attended a NOVA session. They had only had tentative discussion with the education 
faculty member who was the only original NOVA team member. At another school, only 
one member had attended the NOVA workshop and had returned to promote this to other 
faculty and felt he was making progress. In both cases, the faculty had under-developed 
conceptions of action research. So, the absence of consolidated teams in these cases has 
hindered progress to using new approaches and in implementing action research projects. 

In only one instance (5%) was the observed lesson completely didactic, with the 
instructor reviewing homework, with some question and answer participation. This 
instructor had an excellent understanding of the many ways in which children solve 
mathematical problems and wanted future teachers to be aware of these and insisted they 
learn how to solve problems in alternative, but very rigid, ways. Thus, the essence of 
constructivism in which people solve problems in personal ways was reduced to new 
forms of algorithms. 

Overall this data shows that the majority of faculty have shifted their practice to 
more student-centered approaches and that others are in transition. Only one person 
adopted a teacher-centered approach even though in the interview she stated she did 
otherwise. The contradiction between espoused philosophy and actual practice needs 
further exploration. The data from classroom observations supports the notion that action 
research projects, together with the professional development provided by NOVA, and 
well functioning teams at institutions are requirements for effective implementation of new 
models of teaching and learning in higher education. Further analysis of the data is needed 
to examine the relationship between effective action research proposals, teaching efficacy 
and the implementation of inquiry-based lessons. 
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Discussion 

Approximately two thirds of the proposals presented articulate statements about the 
role and purpose of action research. But, given that only a quarter of the proposals were 
scored in the distinguished to proficient range, it is clear that additional work is needed to 
support faculty in developing (and implementing) comprehensive action research 
proposals. There is a need to provide more input on instrumentation for data gathering and 
on data analysis, particularly with respect to qualitative techniques. In general faculty are 
comfortable with quantitative approaches, but less familiar with techniques for gathering 
and analyzing qualitative data. For instance, Kember and McKay (1997) noted that the use 
of control groups is unusual in action research groups, but this was the dominant 
conception of research design in the NOVA action research projects. That faculty are more 
prone to using quantitative techniques suggests that it may be difficult to move faculty in 
science and mathematics disciplines out to the research paradigm with which they are most 
familiar and comfortable. 

The use of a broader range of methodological approaches, as promoted by Keller 
(1998) is likely to provide a richer and more in depth platform for examining teaching and 
learning. Evidence in the research proposals and exemplary cases described shows that 
there is a move towards this in more than half of the cases, where a range of strategies 
have been used by science and mathematics professors to assess student attitudes, 
perceptions and conceptual knowledge. Evidence presented also shows that successful 
implementation of an inquiry approach to teaching was dependent on the successful 
functioning of interdisciplinary teams. The inclusion of education faculty, who are likely 
to be more sympathetic to qualitative techniques, has aided in this emergence of alternative 
assessment strategies in science classes. Despite this positive move toward innovative 
ways of assessing students, there is less evidence on the extent to which alternative 
methods have been used to examine the effectiveness of particular approaches to teaching, 
to reflect on classroom relationships, to consider issues relating to classroom climate, to 
examine personal constructs and their impact on teaching, or even the impact of different 
assessment techniques. Research is primarily reserved for examining the impact on 
student learning and it might be argued that this has been done to justify the introduction 
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of a new course and to counter any negative perceptions about innovation from other 
faculty or administrators. 

There is a substantial body of research on the reciprocal relationship between 
students’ prior experiences, their approaches to learning, their perceptions of the learning 
situation, and their learning outcomes (e.g. Prosser & Trigwell, 1999). For instance, 
students tend to adopt a ‘surface approach’ to learning by memorizing facts if they 
perceive that the environment requires it. In contrast they will adopt a ‘deep approach’ to 
learning if the teaching context demands that they make conceptual connections and that 
they strive for understanding. Prosser & Trigwell (1999) added that there is far less 
research into university teacher’s conceptions of teaching, and “even less into their 
perceptions of the teaching context, their approaches to teaching, outcomes of teaching 
and relations between these aspects of the experience of teaching” (p. 21). However, it 
may be difficult to persuade faculty to engage in this kind of research because it is outside 
of their realm of familiarity. Schon (1995) described the paradoxical case of a professor at 
MIT who had the opportunity to research the impact of computer aided instruction on 
student learning, but who was not able to develop a suitable research question. He 
concluded that introducing and legitimizing action research may prove equally difficult 
with other scholars who would be undertaking the research. More work is needed in the 
NOVA project to encourage faculty to move towards examining their own teaching 
practices within an action research mode that utilizes the pluralistic research methods 
suggested by Keller (1998). 

Action research involves cycles of planning, acting, observing and reflecting 
(Hopkins, 1993). The exemplary projects illustrate how faculty have been through these 
stages once, but there is little evidence of the continuation into additional cycles. 
Reflection necessitates the origination of new hypotheses and research questions for 
ongoing planning, action, observation and reflection. Action research is an iterative 
process. One case at Brescia University shows how one instructor modified instruction in 
the next course offering to account for any negative student perceptions and poor 
conceptual understanding in certain areas of the content, but other than this there is little 
evidence of the cyclical change implied in action research. Further data needs to be 
gathered to establish the extent to which ongoing cycles are being implemented and 
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NOVA faculty should be encouraged to see change as an ongoing process of seeking 
improvement in teaching and learning. 

That professors appear to be less able or willing to use alternative methods for 
personal evaluation and self-reflection supports the analysis that there is a gap in higher 
education between research and practice (Kezar, 2000). It is possible that faculty go 
through different stages as they grapple with how to conduct action research. Initially they 
may find it easier to focus outside of themselves and on students. They may need far more 
support and input, such as through the involvement of a ‘critical friend’ (Kember, 1998) or 
participant observer, to assist them in moving towards research, reflection and analysis on 
their own practice. Kember (1998) described the role of the critical friend in the Action 
Learning Project, an inter-institutional project involving eight institutions in Hong Kong. 
Within the project there were a number of coordinators who worked with ten to twelve 
teams (and there were over 50 teams) in the role of ‘critical friend’. Their role included 
that of (a) ‘rapport builder’, (b) ‘coffee maker’ (ongoing facilitator'), (c) ‘mirror’, (d) 
teaching consultant (e) evaluation and research advisor, (f) writing consultant and (g) 
match maker (putting people in touch with others). NOVA does not have any full time 
staff and it will be a challenge to find ways to promote this notion. 

Professional development is not a one-off event and involves (a) a clear focus on 
learning and learners, (b) an emphasis on the individual and organizational change, (c) 
small changes guided by a grand vision, and (d) ongoing professional development that is 
procedurally embedded (Guskey, 2000). Action research is one way to achieve this and 
this research shows that most faculty in the program have a clear focus on learners and 
learning and have embarked upon change processes at the personal and institutional level 
by implementing these new courses and action research projects. The action research has 
enabled faculty to engage in thinking about student learning and to reflect upon their own 
teaching styles. All proposals espouse the desire to move towards inquiry approaches in 
science and mathematics teaching in higher education and this was observed more then 
two thirds of the classroom observations. Thus, the action research projects have 
contributed to faculty development. 

The Carnegie Foundation for the Advancement of Teaching suggests three criteria 
for defining the scholarship of teaching and learning, (a) that teaching is teaching is deeply 
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embedded in the discipline, (b) that it is an aspect of practice, and (c) that it is 
characterized by a transformational agenda (Hutchings, 2000). The majority of the project 
proposals, cases and exemplars described in this article meet the first two criteria, although 
different action research projects are clearly in different stages of development and 
implementation. First, the action research designs proposed and used for inquiry are rooted 
in the scientific method as evidenced by the large number of proposals that adopted a pre- 
test post-test design, together with control and experimental groups in some instances. 
Therefore, their inquiry is rooted in the particular tools of their discipline. Second, all of 
the cases involve faculty using new approaches to teaching and grappling to understand 
the impact on student learning, that is, the faculty are considering aspects of their teaching 
practice. 

In considering the third criterion, it is important to consider the purpose of the 
research proposals. Within emancipatory action research practitioners work together as a 
group and collectively identify problems and possible solutions (Carr & Kemmis, 1986). 
There is a concern for political change, consciousness raising and the generation of new 
theories, as much as for practical improvements. Most of the action research proposals in 
the NOVA program are concerned with practical issues, like how to improve student 
attitudes and concepts. These are not transformational agendas and so more research is 
needed particularly on the exemplary action research projects that have produced results, 
to establish the extent to which action research projects have transformed personal and 
institutional practices. There is evidence from interviews with administrators in 
universities with NOVA courses that this has happened, but these have not been the focus 
of action research agendas. So, action research projects may still be in transition towards 
transformational agendas. Despite this, the NOVA program can be considered to be 
making a substantive contribution to the scholarship of teaching and learning. 
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Appendix 1 
Scoring rubric 



Score 


Explanation 


1 - novice 


The item is not included. 


2 - apprentice 


The item can be inferred from the text or there 
is evidence of intent that is not elaborated. 


3 - proficient 


The item is stated in the text but is not 
elaborated or is explained in part. 


4 - distinguished 


The item is fully elaborated in the text. 



Scoring dimensions 





1 


2 


3 


4 


Action research proposal 




















1 . The rationale of the study is clearly presented 










2. The purpose of the study is provided adequately 










3. The action research cycle (PAOR) is evident 










4. There are clear research questions or hypotheses 










5. The research methods are clearly stated 










6. Research methods are appropriate to the questions 










7. Instrumentation is described 










8. States who the program participants are 










9. Data sources are identified 










10. Potential sample sizes are noted 










11. Triangulation of data is proposed 










12. States who will collect the data 










13. Methods of data analysis are proposed 










14. Methods of data analysis are appropriate 
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